First Principles Studies on 3-Dimentional Strong Topological Insulators: 612X63, Bi2Se3 and Sb2Te3 
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BiiSes, 819X63 and Sb^Tej compounds are recently predicted to be 3-dlmentional (3D) strong topological 
Insulators. In this paper, based on ab-initio calculations, we study in detail the topological nature and the surface 
states of this family compounds. The penetration depth and the spin-resolved Fermi surfaces of the surface states 
will be analyzed. We will also present an procedure, from which highly accurate effective Hamiltonian can be 
constructed, based on projected atomic Wannier functions (which keep the symmetries of the systems). Such 
Hamiltonian can be used to study the semi-infinite systems or slab type supercells efficiently. Finally, we discuss 
the 3D topological phase transition in Sb2(Tei_vSej)3 alloy system. 
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I. INTRODUCTION 

The topological insulators (TIs)'"^, are interesting not only 
because of its fundamental importance but also because of its 
great potential for future applications^—. TI is a new state of 
quantum matter, and is distinct from simple metal or insulator 
in the sense that its bulk is insulating (with a bulk band gap), 
while its surface (or edge) is metallic due to the presence of 
gapless surface (edge) states. Those surface states are spin 
splitted but with double degeneracy at the Dirac point, which 
is protected by the time-reversal symmetry. The surface states, 
consisting of odd number of massless Dirac cones, are robust 
against time-reversal-invariant perturbations and also are very 
different with Graphene where spin degeneracy is reserved 
(the concept of pseudo-spin describing the sub-lattice degen- 
eracy is used for Graphene). At the early stages of studies for 
TIs, due to the absence of realistic materials, most of the dis- 
cussions were based on the model Hamiltonians'"^. However, 
within last couple of years, this field is strongly motivated by 
the discoveries of several real compounds. 

Two-dimension (2D) TI (or called quantum spin Hall insu- 
lator) has been proposed and realized in HgTe/CdTe quantum 
well structure through varying the well thickness^nli. Three- 
dimension (3D) topological nature of Bii_ ^Sbx alloy has been 
demonstrated by fine tuning of the alloy concentration'^"'"^. 
In both cases, the bulk band gap is of the order of tens meV, 
too small for the potential applications. In our recent studies, 
however, we predicted that Bi2Se3 family compounds'^ (i.e., 
Bi2Te3, Bi2Se3 and Sb2Te3), which are well-known thermo- 
electric materialsi^— , are strong topological insulators with 
surface states consisting of single Dirac cone at the T point. 
Particularly, for Bi2Se3, a bulk band gap of 0.3 eV is pre- 
dicted, much larger than the energy scale of room tempera- 
ture. The compounds are stoichiometric, chemically very sta- 
ble, and easy to be synthesized, and yet its surface states are 
very simple. The existence of such surface states in this family 
compounds have been experimentally confirmed on samples 
ranging from bulk to thin filmi2i2i. 

More and more studies are now going on for Bi2Se3 fam- 
ily compounds, quantitative studies and comparison with ex- 
periments become more and more important. On the other 
hand, fully self-consistent ab initio calculations are quite lim- 



ited due to the large system size. In order to study both the 
bulk and the surface behavior simultaneously, which is neces- 
sary to identify its topological nature, a large slab type super- 
cell is needed, and heavy calculations are involved. To sim- 
plify future studies, it is therefore desirable to have a highly 
accurate effective Hamiltonian, which will be one of the main 
purposes of this paper. It's well-known that the maximally 
localized Wannier functions (MLWF) method introduced by 
Marzari and his coworkers^^'^^ has played an important role 
in constructing the effective Hamiltonian. However, the algo- 
rithm involved does not hold the symmetry when the MLWF 
are constructed from Bloch functions. To solve this prob- 
lem, we introduce a set of projected atomic Wannier func- 
tions (PAWF), whose symmetry can be easily controlled, yet 
keeping high accuracy. With this set of PAWF basis, highly 
accurate effective Hamiltonian can be constructed. Using this 
Hamiltonian, we will demonstrate that the topological nature 
as well as the details of surface states can be all well repro- 
duced. 

The organization of the present paper is as follows. In sec- 
tion II, we describe the crystal structure and symmetries of 
Bi2Se3 family compounds. Section III is devoted to the con- 
struction of the effective Hamiltonian in the basis of PAWF. In 
section IV, this Hamiltonian is applied to Bi2Se3 system, and 
the topological nature and surface properties are analyzed. In 
section V, we discuss the 3D topological phase transition by 
doping Se into Sb2Te3. Final remarks are summarized in sec- 
tion VI. 



II. CRYSTAL STRUCTURE AND SYMMETRIES 

The Bi2Se3 family compounds have the rhombohedral crys- 
tal structure with space group D^^ (R3in), we take Bi2Se3 as 
an example in the following. As shown in Fig.fTfa), the system 
has a layered structure with five atomic layers as a basic unit 
(cell), named a quintuple layer (QL). The inter-layer bond- 
ing within the QLs is strong because of the dominant covalent 
character, but the bonding between the QLs is much weaker 
due to the van der Walls type interaction. The binary (with 
twofold rotation symmetry), bisectrix (appearing in the reflec- 
tion plane), and trigonal (with threefold rotation symmetry) 
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FIG. 1: (Color online) Crystal structure of Bi2Se3 family com- 
pounds, (a) The hexagonal supercell containing 15 atomic 
layers and primitive translation vectors ti_2.3 (b) The top 
view of a quintuple layer (QL) in the triangle lattice. Three 
sets of different sites, labeled as A, B, C sublattices respec- 
tively, are presented. Owing to the symmetry, the stack- 
ing of atomic layers along the z direction is in the order of 
•••-C(Sel)-A(Sel)-B(Bil)-C(Se2)-A(Bil)-B(Sel)-C(Sel)---. 
(c) The first Brillouin zone (BZ). Four nonequivalent time-reversal- 
invariant-momentum (TRIM) points F (0, 0, 0), L(n, 0, 0), F(7t, k, 0), 
and Z(n, n, n) are denoted in the 3D BZ. The corresponding surface 
2D BZ is represented by the dashed blue hexagon, and F, M, K are 
the corresponding TRIM special k points in the surface BZ. 

axes are taken as x, y, z axes respectively, and the primitive 
translation vectors ti,2,3 shown in Fig.[T]are 

ti = (-fl/2,- V3fl/6,c/3), 

ii = (a/2,-^/3a/6,c/3), (1) 

t3 = (0, V3fl/3,c/3). 

Here, a and c are lattice constants in hexagonal cell. The cor- 
responding reciprocal vectors Si,2,3, defined by Sj ■ tj = Indij, 
are given as, 

51 = (-1,- V3/3,/7)/z, 

52 = (l,-V3/3,fe)/i, (2) 

53 = (0,2 V3/3,/7)/z, 

with 

b - a/c, 

h — In/a. (3) 

As shown in Fig. \lla), we take Se2 to be at origin (0, 0, 
0), then two Bi sites are at {+fi, +fi, +fj.), and two Sel are at 
(+v, +v, ±v), defined in the unit of primitive translation vec- 
tors. All the experimental lattice parameters and internal pa- 
rameters fi and V are listed in Table U Fig.IIJc) shows the 3D 
first Brillouin zone (BZ) and the 2D surface BZ of Bi2Se3. 
r(0, 0, 0), L(7T, 0, 0), F(7i, 7T, 0) and Z(n, n, n) are four time- 
reversal invariant momentum (TRIM) points in the 3D BZ. 



TABLE I: Experimental lattice parameters and the internal parame- 
ters of Bi2Se3 family compounds^'. 







SboTej 


BijTe, 


BizSej 


Lattice constant 


a (A) 


4.250 


4.383 


4.138 




c(A) 


30.35 


30.487 


28.64 


inner coordinates 




0.400 


0.400 


0.399 




V 


0.211 


0.212 


0.206 



r(0, 0, 0) and Z(;r, n, n) are projected as F, and Lin, 0, 0) and 
F{n, 7T, 0) are projected as M in the surface BZ. For the choice 
of our cell, Bi2Se3 has the inversion symmetry with inversion 
center at Se2. The space group R3in can be constructed from 
three symmetry generators: I (inversion), cj,- (three fold ro- 
tation around z), and cr^ (mirror plane with its normal along 

X). 



III. EFFECTIVE HAMILTONIAN FROM PAWF 

The construction of the PAWF basis and the effective 
Hamiltonian is a post-process of ab initio calculations. The 
fully self-consistent ab initio calculations are performed for 
the bulk compounds in the framework of density functional 
theory (DFT)2^ using BSTATE (Beijing Simulation Tool for 
Atom Technology) package^** with the plane-wave ultra-soft 
pseudo-potential method^". Generalized gradient approxima- 
tion (GGA) of the Perdew-Burke-Ernzerhof (PBE) type is 
used for the exchange-correlation potential^'. To guarantee 
the convergence, we use 340 eV for the cut-off energy of 
wave-function expansion, and 10x10x10 K-points mesh in 
the BZ. All structure parameters of BiaSej, Sb2Te3, Bi2Te3 
are chosen from experimental data listed in Table U 

After completing the self-consistent ab initio calculations, 
two steps are followed to construct the PAWF basis and the 
effective Hamiltonian. First we disentangle an isolated group 
of bands through minimizing the invariant part of the spread 
functional of wave functions (Q/)^*". In the second step, the 
Hilbert space of this isolated group of bands is directly pro- 
jected onto atomic p orbitals (keeping the angular symme- 
try). Finally a set of unitary rotation matrices ^/(k) can be 
obtained, which can be straightly used to construct the PAWF 
and the effective Hamiltonian. The details are elucidated in 
the following. 



A. Extracting of isolated group of bands 

For a simple band insulator, the disentangling procedure is 
typically not necessary, and it is straightforward to choose a 
set of Wannier basis expanding the identical space of occu- 
pied Bloch states. However, in our case, aiming to describe 
the electronic characters near the Fermi level, both occupied 
and unoccupied bands are necessary, which are not detached 
from higher bands. By partial density of states (PDOS) analy- 
sis, we know that the states near the Fermi level mainly come 
from the contribution of p orbitals of both Bi and Se atoms. 
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Since each atom has three p orbitals, the total number of or- 
bitals of interest is, N - 15, consisting of 9 valence bands 
and 6 unoccupied bands. Following the procedure described 
in MLWF^^, we introduce a big enough outer energy window 
(-20 eV, 20 eV), and a small inner windows (-6 eV, 2 eV). 
A smooth subspace (of N bands) can then be constructed by 
optimizing the spread functional, namely Q/, as suggested in 
Ref.[2^]. 



effective Hamiltonian can be symmetrized, while keeping the 
necessary accuracy. Second, the atomic spin-orbit coupling 
(SOC) can be easily implemented. Since the WF are con- 
structed from atomic orbitals, they are already quite localized 
although not maximally localized by definition. The PAWF 
eff'ective Hamiltonian is expected to be useful for large su- 
percell calculations without extra errors from the symmetry 
problem. 



B. PAWF basis and efTective Hamiltonian 



C. The spin-orbit coupling 



The Bloch functions i^„,k of the isolated group of bands 
are directly projected onto the atomic p orbitals, and the uni- 
tary rotation matrices UiV) can be obtained by proper or- 
thonormalization?^'-'^. With these unitary rotation matrices 
1/(k), we can obtain well-localized PAWF by rotating the 
original Bloch functions in the following way: 



llAnk >= ^ 1/,„„(k)|l/'„,k >, 



(4) 



The SOC is important for the topological nature of Bi2Se3 
family compounds, and should be included in all analysis. 
Since SOC is mostly local atomic physics and has little k- 
dependence, the simplest way to supplement SOC on top of 
our PAWF effective Hamiltonian (denoted as from now on) 
is to include an additional local term in the total Hamil- 



tonian^, 



(9) 



where 



(5) 



where n and m-l,2 ■ ■ ■ are band index, Nk is the total num- 
ber of fc-points, denotes the n-th PAWF orbital, which is 
centered at the lattice vector R. 

Using the PAWF as basis, the effective Hamiltonian H^(k) 
can be obtained correspondingly by rotating the original 
Hamiltonian, 



: 'itH{\.)nj. 



(6) 



Because our PAWF are constructed from atomic orbitals, 
which have atomic symmetry, we can further symmetrize the 
Hamiltonian using their atomic characters. Using Fourier 
transformation, the PAWF effective Hamiltonian in real space 
can be given by 



//W(R) = ± y 



e-''«//^(k). 



(7) 



The Hamiltonians at any other A:-point can be now obtained by 
transforming H^iW) back into k space. 



//"'(k) = ^e"'«//^R). 



(8) 



All the hopping matrix elements in the effective Hamilto- 
nian are directly calculated via constructing the PAWF. This 
approach differs a lot from the conventional Slater-Koster TB 
method^^ where hopping integrals up to certain neighbors are 
obtained by a fitting method. The present method has several 
special advantages. First, it is not necessary to maximize the 
localization of Wannier functions (WF). Such step will usually 
break the local symmetry of WF. In our case, the WF could be 
constrained to satisfy local crystal symmetry, and the PAWF 



(niAm^c^){W X P] ■ 0-. 



(10) 



H""'^ comes from the SOC interaction, while V and cr repre- 
sent total potential and Pauli spin matrices respectively. 

As discussed above, our PAWF are constructed from atomic 
p orbitals (having atomic angular symmetry), the SOC Hamil- 
tonian can be straightforwardly written down, in terms of 
those PAWF basis, 

\P., T>, \Py, T>, \Pz, T>, \P., i>, \Py, i>, \Pz, i>, (11) 

where f (J.) indicates the spin. With this basis set for each 
atom, //""^ part can be expressed as 



-/ 1 

/ -/ 

^ -1 /0 

2 -1 /0 

-/ -/ 

1 / 



(12) 



where A denotes the SOC parameter. We take the SOC pa- 
rameters of Bi, Se, Sb and Te atoms from Wittel's spectral 
data {Abi = 1.25 eV, Ase = 0.22 eV, Asb = 0.4 eV, Aje = 0.49 
eV)^. 



D. Accuracy of effective Hamiltonian: results for bulk 

To demonstrate the quality of the PAWF effective Hamil- 
tonian, here we compare several important properties calcu- 
lated from the PAWF Hamiltonian with that obtained from the 
ab initio calculations. The first property to be compared is the 
the calculated bulk band structure. As shown in Fig.|2a), for 
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FIG. 2: (Color online) The comparison of PAWF bands (red dashed lines) with the ab initio band structures (black solid lines) of Bi2Se3. (a) 
The results without SOC. The PAWF effective Hamiltonian almost exactly reproduces the ab initio band structures. (b)The results with SOC 
for BiaSes. 




FIG. 3: (Color online) The comparison of PAWF bands (red dashed lines) with the ab initio band structures (black solid lines) of (a) Bi2Te3 
and (b) Sb2Te3. Only the results after including the SOC interaction are shown here. 



TABLE II: The parity eigenvalues of the energy bands of Bi2Se3 fam- 
ily compounds at F point, obtained from the effective Hamiltonian. 
The parity numbers of nine occupied bands and the first conduction 
band are shown (the corresponding band energy increases from left 
to right). The parity products of nine occupied bands are given in 
brackets. 



Parity 


Bi2Se3 


(-1) 


-1 


1 -1 -1 


1 1 


-1 


-1 


1; 


-1 


Bi2Te3 


(-1) 


-1 


1 1 -1 


-1 1 


-1 


-1 


1; 


-1 


SbzTes 


(-1) 


1 


-1 -1 1 


-1 1 


-1 


-1 


1; 


-1 



Bi2Se3 without SOC, the band structures obtained from PAWF 
effective Hamiltonian can almost exactly reproduce the ab ini- 
tio band structures. After taking into account the SOC inter- 
action (as shown in Fig.|2b)), the quality is slightly reduced, 
but it is still reasonably good since only the k-independent 
SOC interaction is implemented in the effective Hamiltonian. 
For Bi2Te3 and Sb2Te3, the PAWF effective Hamiltonian show 
similar quality (shown in Fig.[3la)-(b)). 

Parity is another important quantity to distinguish the topo- 
logical characters of Bi2Se3 family compounds. Because the 
present systems posses the inversion symmetry, the method 



proposed by Fu and Kane- is used here. For all three sys- 
tems, we calculate the parity eigenvalues of nine occupied 
bands and the first conduction band at the time-reversal in- 
variant momentum (TRIM) points F, L, F, and Z. The results 
for the r point, obtained from the effective Hamiltonian, are 
listed in Table [III As can be clearly seen, the parity products 
of occupied bands are -1 for the F points. For the other TRIM 
points, L, F, and Z, the detailed numbers are not listed here, 
but our results give +1 for three of the compounds. From 
the parity numbers, we can therefore identify the correspond- 
ing Z2 invariantsiii^ as 1 for all the three compounds. This 
means they have nontrivial topological nature, in good agree- 
ment with our previous studies'^. From above comparisons, 
we conclude that the quality of the PAWF effective Hamilto- 
nian is as good as that obtained from the ab initio calculations. 

IV. THE PROPERTIES OF SURFACE STATES 

Once the effective Hamiltonian has been constructed from 
above procedures, we will be able to calculate the surface 
states of semi-infinite systems from the iterative Green's func- 
tion method^*, as described in our previous studies'^. The ex- 
istence of gapless spin-filter surface states is the direct man- 
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r K Depth (QL) Depth (QL) 



FIG. 4: (Color online) (a) The calculated band structures of Bi2Se3 slab with film thickness of 25-QLs. The bands along the F — > A" direction 
are shown. The Fermi level is located at energy zero. The two surface bands in the bulk gap (around the F point) are denoted as Si and E2. (b) 
and (c) The real-space distribution of eigen wave-functions (\ifi„k{r)\') for different ^:-points (k], and as indicated in (a)), (b) or (c) are for 
A:-points along the Ei or E2 lines respectively. The inset of (c) is the zoomed-in picture for the ki point eigen wave function, where the atomic 
layer resolution is visible. 




r M Depth (QL) Depth (QL) 



FIG. 5: (Color online) The same as Fig.U but for bands along the F — > M direction. 



ifestation of the topological nature. The crystal structure of 
Bi2Se3 family compounds can be understood as the stacking 
of QLs along the z direction. The inter-layer bonding be- 
tween two QLs is much weaker than that inside the QL, it 
is natural to expect that the cleavage plane should be between 
two QLs. This fact has been well confirmed by the recent 
experiments on the layer-by-layer MBE growth of ultra-thin 
film^^. We therefore focus our studies on this type of sur- 
face termination (with Sel atomic layer as the top most layer). 
Our previous studies have shown that gapless surface states 
with single Dirac point (located at F) exist for all three com- 
pounds, BiiSes, Bi2Te3 and Sb2Te3, but not for Sb2Se3. The 
dispersion of surface states around Dirac point is highly linear, 
and these surface states can be described by a simple contin- 
uous model'''. Here, we will not repeat those results, on the 
other hand however, will focus on some other details of sur- 
face states, including the penetration depth, the spin-resolved 
Fermi surface, and the chiral spin texture of the surface states. 



A. The penetration depth of surface states 

The spread of surface states, or in other words its spacial 
penetration depth into the bulk, is an essential quantity for the 
potential applications of these surface states if any. In order to 
clarify this point, a free standing slab model is constructed us- 
ing our PAWF effective Hamiltonian. We use a slab consisting 
of 25 QLs, which is thick enough to avoid the direct coupling 



between the two surfaces, i.e, the top and the bottom surfaces 
of the slab. In order to make the calculations more realistic, 
we further take into account the surface correction. The bulk 
PAWF effective Hamiltonian is used to construct the Hamil- 
tonian of the slab, however for those layers close to the sur- 
face, the on-site energies of PAWF should be modified from 
its bulk counterparts due to the presence of vacuum (i.e, the 
surface potential). By comparing with the fully self-consistent 
ab initio calculations of thin slab, all the correction of on-site 
energies for PAWF {AE„ - £^^"''f"" _ /jj™'*) can be obtained. 

The calculated band structures of our 25-QLs Bi2Se3 slab 
along r — > and F — > M lines are shown in Fig.|4]and Fig.|5] 
Clearly, within the bulk energy gap, there exist two surface 
bands (marked by 1.\ and Y.2), which are degenerate at F point 
(where so called Dirac cone exists). The two surface bands 
(Zi and E2) sampled by several ^-points (with A:,, i= 1,2,3). 
The real space distribution of eigen wave functions for those 
sampling A;-points are plotted in Fig.|4fb)-(c) and Fig.|5lb)-(c) 
respectively. For those ^-points close to the Dirac point (such 
as k\ and ^2), the distribution of wave functions are very lo- 
calized to the surface region, with a typical spread of about 
2 QLs (about 2nm in thickness). By moving away from the 
Dirac point, the penetration depth increases, and finally for 
kT, point, where the surface state almost merges with the bulk 
states, the eigen wave function becomes a extended state. For 
both Y - M and Y - K lines, the same behaviors are observed. 
To further see the oscillating behavior of surface state distri- 
bution over atomic layers, we plot the distribution for k\ in 



vacuum 




FIG. 6: (Color online) The spatial charge distribution within an en- 
ergy window of lOmeV around the Fermi level. The top atomic layer 
means the surface of the slab. It is clearly seen that electrons are 
mainly localized at the surface of the slab. 

the inset. Finally, we can conclude that surface states are very 
localized to the surface region, and the penetration depth is 
about 2 or 3 QLs. 

To make further comparison, we have performed a fully 
self-consistent ab initio calculations for a 3-QLs slab. The 
real space charge distribution of surface states can thus be ob- 
tained, by integrating the states within an energy window of 
10 meV around the Dirac point. As shown in Fig.|6] the charge 
distribution is really localized at the surface region. Further 
more the charges are not centered at the atom, but rather dis- 
tributed mostly between the Bi and Se atoms, where strong 
covalent bonding is expected. 



B. The spin-resolved Fermi surface 

Probing the k Berry phase enclosed by the Fermi surface 
of surface states is one of the most direct methods to distin- 
guish TI. Now we will analyze the spin-resolved Fermi sur- 
faces around the Dirac point of the semi-infinite BiaSes sys- 
tem. With the Green's functions calculated from the effective 
Hamiltonian, the spin-filter surface states and the correspond- 
ing Fermi surfaces can be obtained directly. 

As shown in Fig. |7] when the Fermi level is close to the 
Dirac point, the corresponding Fermi surface is nearly a per- 
fect circle; while if the Fermi level is away from the Dirac 
point, the properties of the surface states are significantly af- 
fected by the buUc states and thus satisfy the crystal symme- 
try. For example, the Fermi surface for £'y=0.1eV (shown in 
Fig.|2fa)) looks like a small circle, while the Fermi surface for 
£'/=0.25eV (shown in Fig.|3b)) looks like a hexagon satisfy- 
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FIG. 7: (Color online) The spin resolved Fermi surface of surface 
states, (a) and (b) show the Fermi surfaces for Fermi energy at 0. leV 
and 0.25eV, respectively. The Fermi surfaces are denoted by red cir- 
cle, and the in-plane spin orientation is indicated by green arrows. 




K r M Energy(eV) 



FIG. 8: (a) Energy bands of a 3-QLs slab, (b) The corresponding 
density of states (DOS). The DOS near the Fermi level is highly lin- 
ear to energy due to the presence of Dirac cone type surface states. 



ing C3. symmetry. The spin orientation for states around the 
Fermi surface is marked as well by arrows. The magnitude 
of the spin z component is very small, only about 1/40 of the 
in-plane component. Thus the spin almost completely lies in 
the plane. Moving around the Fermi surface, the spin orien- 
tation rotate simultaneously, forming a spin-orbit ring, which 
carries a n Berry phase. This signifies the topological nontriv- 
ial properties of Bi2Se3. In addition, as shown in Fig.|2l the 
spin orientation of the ring belongs to the left chirality (the 
normal direction of the semi-infinite system is defined as z di- 
rection) which is again one of the important manifestations of 
nontrivial topological characters. 



C. The linear dispersion of surface states 

The gapless surface states that connect the bulk valence and 
conduction bands have almost linear energy dependence near 
the Dirac point. Linear band structure in 2D should lead to 
linear density of states (DOS). To be specific, in the bulk en- 
ergy gap, the energy bands are mostly from the surface states, 
so the DOS in the bulk energy gap are expected to be highly 
linear. Fig.lSfa) shows the band structures of a 3-QLs slab. Its 
coiTesponding DOS (presented in Fig. |8jb)) shows very nice 
linear energy dependence within the bulk energy gap as ex- 
pected. This type of DOS can be easily measured by low tem- 
perature scanning tunneling spectroscopy (STS), which will 
provide an indirect method to probe the existence of linear 
surface states. 



7 




X 



FIG. 9: (Color online)The Gap energy of Sb2(Tei_ vSej:)3 as a func- 
tion of Se concentration x. TI denotes the topological insulator, while 
SI means the simple insulator. 

V. 3D TOPOLOGICAL PHASE TRANSITION 

The above discussions presented in sections II and III con- 
centrate on the effective Hamiltonian and the detailed under- 
standing to the properties of surface states. By such a way, 
we can analyze the topological nature of specific compound. 
On the other hand, however, the topological nature can be 
also understood from simple bulk studies, where a gap close- 
reopening transition (a topological phase transition) can be 
obtained by tuning some parameters'^. We will show in this 
section that such a phase transition can be indeed obtained in 
Sb2(Tei_vSef)3 alloy system. 

The virtual crystal approximation (VCA) method proposed 
by Bellaiche'^ is used here to simulate the doping process. 
The solid solution elements are treated as different atomic 
species with their own weights, rather than being treated as 
a whole virtual atom. With this method, we investigate the Se 
doping dependence of band structure of Sb2(Tei_ jSe t)3. We 
assume that crystal parameters and positions of inner atoms 
change linearly with doping, and they are obtained by linear 



interpolation in calculation. The evolution of the gap energy 
of Sb2(Tei_jSej)3 as a function of Se concentration is calcu- 
lated and illustrated in Fig.|9l With increasing Se doping, the 
valence band maximum and conduction band minimum get 
closer gradually (0 < x < 0.94), attributed to the gradually 
decreased SOC strength. Consequently, the two bands cross 
at the critical point x - 0.94, resulting in a 3D topological 
quantum phase transition. An band-inverse appears when the 
doping concentration further increases. Therefore we can con- 
clude that the occurrence of 3D topological phase transition is 
driven by the SOC interaction. In reality, the most stable crys- 
tal structure of Sb2Se3 is slightly distorted from the rhombo- 
hedral structure of Sb2Te3, however, as long as the Sb2Se3 
is topologically trivial, such 3D topological phase transition 
will be expected. At the phase transition point (x=0.94 as pre- 
dicted in our calculations), the 3D Dirac cone may be expected 
in the bulk band structure^. 

VI. CONCLUSIONS 

Bi2Se3, Bi2Te3, and Sb2Te3 systems are a new class of TIs. 
In the present paper, we construct the effective Hamiltonian 
for this family compounds based on the PAWF method. The 
effective Hamiltonian can well reproduce the ab initio band 
structures and its topological nature. The penetration depth 
and the spin-resolve Fermi surfaces of surface states are cal- 
culated and analyzed in detail. At the end of the paper, we 
discuss the 3D topological phase transition in Sb2(Tei_vSei)3 
and suggest that 3D Dirac cone can be obtained by doping Se 
in Sb2Te3. Finally, we hope that our effective Hamiltonian can 
provide further understanding of these materials in future. 
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